The pipeline test will not pass until optimum-habana matches the latest changes in transformers huggingface/transformers#34769, namely the task name image-to-text ==> image-text-to-text in examples/image-to-text/run_pipeline.py for many of the VLMs. I have currently validated the pass using my own test script https://github.com/Spycsh/qwen-vl-hpu/blob/main/qwen2_vl.py.

age-to-text ==> image-text-to-text in examples/image-to-text/run_pipeline.py for many of the VLMs. I have currently validated the pass using my own test script https://github.com/Spycsh/qwen-vl-hpu/blob/main/qwen2_vl.py.

Are you saying we need transformer4.47 for this to work or can you update the examples/images-to-text/run_pipeline to support both cases?

@tthakkal Could you also review this please. THanks.

age-to-text ==> image-text-to-text in examples/image-to-text/run_pipeline.py for many of the VLMs. I have currently validated the pass using my own test script https://github.com/Spycsh/qwen-vl-hpu/blob/main/qwen2_vl.py.

Are you saying we need transformer4.47 for this to work or can you update the examples/images-to-text/run_pipeline to support both cases?

Yes. An update to latest transformers is needed here. run_pipeline.py also need to be updated correspondingly. I will look into this and get back to you later.

Hello, with some small fixes, now the example can be run with following command under transformers 4.45.2 (current optimum-habana compatible version)

python3 run_pipeline.py     --model_name_or_path Qwen/Qwen2-VL-2B-Instruct     --bf16

Kindly review this at your convenience. Thank you! :)

Hi @vidyasiv @jiminha , I have added the tests and README, tested the Qwen2-VL-7b and Qwen2-VL-2b with GAUDI2_CI=1 RUN_SLOW=1 python -m pytest test_image_to_text_example.py -v -s -k Qwen/Qwen2-VL-2B-Instruct , and also fixed the issue when using HPU graph and made HPU graph enabled as default. Now the perf with warmups should be good enough.

Please review at your convenience. Thank you! :)